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40? 


MEMORANDUM FOR: Sayre Stevens 
. Deputy Director for Intelligence ~ 


STAT 


oe 


FROM ; ; ‘ 
Acting Director, Center for Policy Support 


SUBJECT - : Improving Intelligence Analysis 


Aa, 


1. This relates to our discussion on February. 16 
about alternative approaches in intelligence analysis. 
Four of these~-alternative hypothesis analysis, competitive 
‘analysis, devil's advocacy, and alternate conclusions to 
a best judgment--are discussed in the.attached memorandum 
STAT “drafted by[ .- 4 


2. Also attached is a companion piece by[_ sd STAT 
STAT [____—s«djthat examines one means of institutionalizing such 
approaches and augmenting the analytical process. The 
memorandum discusses four. frameworks in which new panels 
of outside specialists could be constructed for the DDI. 


: 3. These memoranda are preliminary cuts at the issue 

* of institutionalizing alternative hypothesis analysis in 
the Directorate. Your reactions to the paper on panels 
will assist us in tage aa the paper requested by the 


DDCI (see attachment) . ; 
: ods 


Attachments: _ tas ; ; 
As stated . _ a _ 2. 


: 4) 


cc: Paul Walsh 
Evan Hineman 
Karl Weber 
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DRAFT 
March 1977 


MEMORANDUM: 


SUBJECT :. Improving ene Quality of Intelligence 
a : Analysis 


Introduction 
1. Intelligence analysts usually work with incom- 


plete data and the deceptive areas of human perceptions 


and intentions. Because of the uncertainties, different 


conclusions can sometimes be drawn from the same data. 


The problem for the intelligence analyst is to ensure that 


the data are rigorously examined, the uncertainties are 


clearly stated and the implications of alternative con- 
clusions are clearly drawn. The jeiriey of iatetiigence 
analysis for the policy eeeenead intelligence consumer 
is in direct proportion to how the analysis meets these 
criteria. ‘ 
2. Four analytical techniques that appear to have 


merit for intelligence analysis include (1) alternative 


hypothesis analysis (2) competitive analysis (3) a devil's 
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advocate or challenging mechanism and (4) alternative 


‘conclusions to a best judgment. 


Alternative Hypot thesis Analysis 


3. ‘Alternative hypothesis is a technique that en- 


sures Chics examination of more than one hypothesis. 


. 


Problems that lend themselves to this kind of treatment 


include those in which alternative hypotheses already 


exist in the community but have not been rigorously 


analyzed; those about Whalen enough significant disagree- 


ment. exists to develop possible alternative ‘hypotheses; 


and those fraught with so many uncertainties that al- 


ternative hypotheses should be developed. Time con- 
straints, the realities of human nature, and past 
experience indicate nat this approach would be most 
effective if individual alternative hypotheses were 
developed by separate 2 one of people. 

4. A supetantivesy strong project monitor, 
preferably one skilled in understanding ane working 


out the human problems likely to be generated by such 


an exercise, would be required. He would.staff the 


' groups, serve as an interface between them, establish 


procedures, ensure that teams have equal access to 


all pertinent data, and generally monitor their progress. 


1 
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. Competitive Analysis 
. 5. In contrast to the alternative hypothesis method, 
the competitive analysis technique does not presume ‘that 
alternative hypotheses already exist or can be developed. 
iistenay to or more groups examine the same problem 
iidesendentiys and they may or may not use different 
“hypotheses or reach different eonowetbnes. Competitive 
analysis already takes place fo some Setane among the 
euea ous government agencies. Policy questions that 
generate wide ranges of uncertainties or a uaetesheaes 
also lend themselves to this kind of treatment. 

7 6. Both competitive groups could be staffed by 
Agency people, or one of the groups could be staffed 
by outside contractors. The latter seems preferable. 
Experience indicates, moreover, that efforts should be 
made to choose people with a mix of views. If either 
team were dominated by one set of preconceived biases, 
significant hypotheses might not be abjaceiweay and 
gigovousiy sidivned. Project monitors would also be 
required for competitive aaeiweus exercises. Their 
functions and required skills would be the sane as 


those discussed above. 
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_ Devil's Advocate/Challenging Mechanism 
| 7, Establishing a devil's advocate or challenging 
function would be a third possible way of improving 
intelligence analysis. Instead of facilitating a positive 
exercise of pursuing various analytical paths, a devil's 
mavecuts function would serve more as a cra trque mecha- 
nism. It could be reserved for those analytical products 
that could profit from an occasional TBEanes erebine to 
determine whether or not analytical methodologies used 
were sound, whether sufficient sicanetbus weweeede 
_ were surfaced, and whether any of the latter had been 
discarded too early in the analytical process. Taking 
an analytical product through such a rigorous test, 
either during this formative Sage or after its completion, 
should give analysts and consumers more confidence in 
the product. , | 

8. This function. Gourd ba werteried by appropriate 
Agency personnel or outside specialists. Selecting the 
latter would have the advantages of keeping the communi- 
cation lines with outside research specialists open and 
guarding agianst an Agency mind-set predominating. 
Specialists could be called in either on a regular pants 
or as needed. . 

Administral » tora ep ely 


-4- 
Approved For Release 2005/11/28 : CIA-RDP86B00985R000100110011-8 


" Approved For élease 2 2005/1 4128, : GIABOP&6B90986R0001 00110011-8 


Alternative Conclusions to a Best Judgment 


9, Presenting alternative conclusions to a best 
_ judgment is another method of, focusing adequate attention 
on alternative hypotheses, and it is fairly self-explana- 
voEye When analyzing data that could easily lend itself 
to various interpretations, an analyst could systematically 
set forth the possible alternate conelusions with their 
individual policy implications. One person could under 
tae ¢his type of approach Bene, or several analysts 
could work together, each Heine tasked to develop ae 
or more of the possible conclusions. 
Utility to Managers 
10. The results of any of the above approaches 
would be of some use to all levels of Agency managers. 
They would eee the benefits of an additional quality 
control system on. their products, knowing that the key 
Sagmente and assumptions ‘of their analysis have been 
| rigorously Peseees There would also be the advantage 
“of being able £6 inform policy oriented consumers of ssh 
unresolved uncertainties and the implications of 
any significant alternative hypotheses. Managers would 
profit by having been alerted in advance to possible 
AR rill isivaliy g = froma & 3g uny 
Ses 
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arguments against best judgments and by being better 
prepared to defend them. Additional data-or new metho- 
dologies might also be surfaced as a result of any of 
these approaches. * 


ll. The advantages discussed above would have to 


be weighed against several arsedvancagey: Overseeing 


any of chess techniques would require senetdenanid 


additional time and energy from Agency managers. A 


professional atmosphere would have to be maintained 


’ in which the focus would not be on the data or the 


Cone Tue Leney but on the analytical process. All 
participants would have to appreciate that our AnaIVEROAI 
work seldom produces a single, unassailable conclusion. 
2. These approaches ee would require additional 
personnel and financial resources. Because such efforts 
would be expensive, they should be focused selectively 


on only those analytical problems that have serious 


policy implications. < 7 ee ry 
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MEMORANDUM 


' SUBJECT: FRAMEWORKS FOR NEW 


DDI PANELS OF OUTSIDE EXPERTS 


Panels of outside experts Sayiee, consult with, produce, 
and critique work for our apadyeice) components. They are 
numerous and multifarious: 


' Oversight and review: PFIAB ., 


Intelligence Oversight Board 


Advisory ' :  §ecientific and Technical 

; - + Rdavisory Panel 

Military-Economic Advisory Panel . 

‘Nuclear Intelligence Panel 

the proposed Intelligence 

Advisory Panel (to consult on 
estimates with 
the NIOs) 


Consultative ; | STAT 
: STAT 


Special and Ad Hoc : "Monster" Panel 
: Laser Weapons Panel 
: Particle Beam Weapons Panel 


These.and other panels and consulting groups employed in 


earlier years have a great deal in common. Membership character- 


-istically is small (averaging about 10 or 12), highly competent 


(leading senior specialists), and diverse. In the cases of all 
the panels working with DDI offices, a single intelligence issue 


or a related set of problems determined the membership. The 


STAT 
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- The "Monster" Panel (inactive now) joined scientific and 


technical people with a variety of skills. - {| STAT 


4 enn 
STAT Wee In contrast, the proposed In- 


‘telligence Advisory Panel would have, a. much larger member 


En ae eee nooner 


ship (about EHDSE: dozen) with no common focus. . Groups of 
estimates and plenum sessions would be rare. 
All of the DDI panels ¢ are Syeiueese highly a people 
in the directorate ‘who work directly with them. They are 
said to make valuable, substantive contributions and to 
provide an important means for analysts to Keep up with 
“new methods and data in their own and other relevant fields. 
The panels have apparently Paatana vital, flexible, and 
non-partisan. Unlike the old ONE group of consultants--many 
of whom had 2ong been out of touch with new Goyer Cnre in 
their fields and with the new. - genexation of sokolscuespaneitecs 
. Tyee by DDI offices are active leaders in their fields. In 
some cases, the usefulness of DDI panels transcends the needs 
of sponsoring offices. STAT 
STAT 7 | 
routinely includes representatives of several DDI 
offices at its meetings; and the BIAS TREY AConOnS Advisory 
Panel impacts on offices gence than’ ‘OSR sand OER. 


The costs of maintaining the various DDI panels apparently : 


are re LAeineeg FomRélbase POtsé 11128 sClAxRDResbboessRoe feradowrArs diem 
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bo consulting fee and travel expenses, and sponsoring offices 


ae incur additional costs in direct support of panels. The 


Latter vary from one panel to another largely according to 

: the amount of paper that panelists receive and the number of 
meetings held. In most cases, remunetation ie sae the main 
factor. that motivaces panelists to assist the agency. Some 
in fact provide their service gratis, and many . command 
higher eonsultine fees in the pervare séctor. DDI consul- 
tants work for the Agency primarily for Sines reasons: the 
Se eaiaiee of doing. so and of being fneiuaed ina small, elite 
group of leaders in their fields; unabashed satriouren: the 
challenge or perhaps the excitement of the task (the latter 
certainly was a main motive for members of the "Monster" | 
Panel). Other ieee altruistic motives undoubtedly are im- 
portant too: the desire to keep up with new data and 
analyses available in the intelligence eommuna ty) and perhaps 
even the desire to keep a covetous eye on new _ technological 


developments speond in éraee to co-opt them. 


FRAMEWORKS FOR NEW DDI PANELS 
Despite the numerous. and successful precedents in the 
scientific, technical, military, ane mi litary-economic Eeeaoey 
other DDI components have done little to take advantage of out 
side expertise on an organized basis. Most of the latter-- 
particularly in regional and political anilyeis-cwese likely 


cauld benefit from consultative relationships. New panels or 
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consulting groups could be set up in four general frameworks: 


A. Directorate Panels 
B. Regional Panels 
Cc. Issues and Problems Panels 


D. Consulting Matrix 


A. Directorate Panels 


A Senior Foreign Intelligence Review Panel of highly 
qualified persons with experience at the top ranks of pub- 
lic and private foreign affairs bureaucracies could perform 
a number of valuable services: 

--assist the DDI and the DCI in evaluating the scope, 

timeliness, policy-relevance, conceptual and 
structural merits of DDI research programs 


-~-review selected finished intelligence products for 
the same purposes 


--advise on new methods, data, and alternative 
hypotheses 


“assist in the production of predictive intelli- 
gence by suggesting different analytical frame- 
works and by warning of otherwise unanticipated 
trends and developments 

--help keep our middle and senior managers (many of 
whom have not kept up their outside ties) better 
in touch with new scholars and developments in 
private research. = tes 

Membership of approximately a dozen most likely would 

be ideal. “Two or more top officials of leading think tanks 
like the Carnegie Endowment or the Georgetown Center 

(which do not perform government contract research) should 
beApinevaddcat Relewse2685/f hah : Che RPPSGRARDSEROAHIDO+AONLBn g , 


business, finance, industry, ete.--should also be included. 


BS cae 
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Costs for such a panel would be minimal. A critical 
difference between such a panel and the others described 
below is that it would impact primarily on ranking DDI 


officials rather than the working level. 


An alternative approach is to establish a larger DDI 
panel to consult and advise on individual and related pro- 
Jeekee If the major disciplines and interests of all the. 
REooueEter offices were represented (including strictly 
regional specialists), a "super" DDI ‘panel would probably need 
between 30 and 50 members. A panel of such proportions would 
be costly, unwieldy, and difficult to manage and convene. 
‘Like the proposed Intelligence Advisory Panel for the NIOs, 
embers could be convened in small groups to advise on par- 
ticular projects. Unlike a senior review panel, a large 
consulting panel would impact primarily with acewing level 
people. Such a panel would be particularly advantageous if 
the NIO panel is not approved, but redundant in the main if 
it is. It is appropriate, to note ee cyeons of the 
Academic Coordinator's directorate committee* are opposed to 


ee Pirsp 


"super" DDI panels in any form. 


oF 
«— 


.B. Regional Panels 


Interdisciplinary regional panels could be established 


in six or more regional areas to assist working level people 


STAT 
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_, throughout the directorate. Regional panels comprised of 
economists, political scientists, historians, sociologists, 
and others would: 


‘-—Provide an interdisciplinary forum to advise 
and critique current and planned research. 


--Advise on methodological innovations and new 
; data bases in the private sector. 


‘.-Tnform our people of the latest research in 
the private sector. 


--Recommend new lines of analysis and alternative 
hypotheses. wee 


~-Alert analysts to emerging trends ana Likely 
future developments of importance in regional 
affairs. 

_ In most cases, regional panels need not have more than six to 
ten members who whould be convened three or four times 
annually and who would be expected to stay in regular contact 
with our regional affairs officers throughout the year. 


Because of the interdisciplinary composition of the panels, 


they would best be administered through the Center for Policy 


>. *¥ Support. Since regional panels would be established pri- 


marily to assist working level analysts, maximum contact 
between a large number of analysts and panelists would be 
expected. | 
C. Issues and Problems Panels 

As explained above, a number of panels already exist in 
the scientific, technical, military, and military-economic 


fields adevoted to specifi Fae aG ems 6£ problems. 
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'. Additional panels in those fields may be desirable now and 
in the future. Such policy issues as non-proliferation, ter- 
rorism, human rights, technology .transfer, narcotics, etc., 
_might be more fully investigated with the assistance 2 panels 
. of outside experts. Some ad hee panels, could be set up tem- 
: hevaniiveand could remain quite small; others would need to 
be larger and more anduriag: . = . 
D. Consulting Matrix 
The approaches qieeuseda above are complementary and can 
be employed simultaneously as long as central seta tacion 
and control is exercised. In such a setting, working grups 
of consultants could be convened from aveecvent panels to 
work together and with analysts on interdisciplinary problems. 
The advantages of a consulting matrix, either in addition 
to or instead of individual panels, are: 
--Would permit maximum flexibility for bring- 
ing diverse viewpoints and backgrounds to bear 
on a problem. 
“e-Would permit the DDI to maintain a large and 
protean stable of consultants in the many 


disciplines. 


--Might well cost less than an equivalent number 
of consultants tied to panels that meet periodically. 


Such an approach would also have significant disadvantages unless 


___.it were tied in closely with other, specialized panels: 


7 
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_T7consulting relationships require sustained 
and regular contact if they are to be ef- 
fective; in a matrix approach not all con- 
Sultants will be regularly involved. 

--central management would be necessary to 
coordinate a consulting matrix, but at’ the | 
expense of needed involvement by line managers 
and analysts. 

CONCLUSIONS 

“More detailed information about the costs of maintaining 
existing panels~-and projections of likely costs of creating 
new ones~-are needed. In ‘some instances, major costs are not 
accounted for because records are not kept of the time spent 
by line personnel in the direct Support of panels. Estimates 
of these--and perhaps other--hidden costs should be made by 
offices now Sponsoring panels and evaluated by the DDI Manage~ 
ment Staff as a corrollary to this study. It should be em- 
phasized in addition, that the largest single expenditure for 
panels frequently is the initial cost of contacting, clearing, 
processing, and setting up the group. Opposition to additional 
panels of outside experts will no doubt arise in at least a 
few offices which seem to prefer informal and Ancoordinated 


methods to keep in touch with outside experts or disdain such 


contacts -attogether.: Reluctant Managers probably will have 


to be shown that Panels can be both inexpensive and highly bene- 


ficial to their research efforts. Better communications within 


the directorate about the experience of existing panels most 
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